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The management and evaluation of education programs 
requires a timely, accurate information system. The discrepancy 
analysis model described aids in providing this information in a 
format that may be processed and yields the method by which such a 
system may be developed and implemented. Building from specific 
statements of expected project performance, this study shoiied hov 
questionnaires may be developed, analysis plans prepared, and 
analytical reports produced to enable a re viewer to quickly learn the 
status of an education project. The system provides two basic types 
of data for decision-makers: basic descriptive information to permit 
statistical summaries of program operation to be prepared and 
reviewed, and operational consistency data which allow 
dec ision- makers to review project operations and to function by 
management-by-ezception, concentrating their efforts on those 
projects most in need of technical assistance. (Author) 
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PREFACE 



The research Which led to the development and insplcanentation of the 
system described In this report was st|)ported in part by Office of 
Education contract mute* (^CH!-0-71-371S. As the Office of fiducaticn 
provide? a nonreatrictlng atmosqihere in which research way occur, the 
conclmJons are those of the author and not necessarily those of the 
Office of jAhication, Special tlvinks go to Drs. Robert Hall and 
Francis Corrignn of the U.S. Ofricc of Education and Mr, Robert Crosby 
of KMC! Re«;o;irch Corpomion for their helpful si^t^tions and criticisms 
throuyjuiirt the stud>% 
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Intr otl uctica i 

TWO basic types of (bcl<:iont; must he ratic with rcgan to operational 
education proftrams and projects— fimdinjj/rrjftmding and project monitoring. 
In order to properly nake these decisions, certain types of infbnaation 
arc nccessan*. Amonj; the ty\)o^ of information required to st^jtort the 
noc^ssar>» dci'i.<'ionii arc* a ilescription of ictiiiil on<;olng project and 
pio.un»»!i activities, the effect of pi^Vunt nml prof.rnni activities, and 
(for ti'ainiiJti pmgr.ims) nfinnos-er r.CfJs and stJppliefs. liach of these 
infomotioii ncecijt must be siipr.ortctl by ar. information sy.«*tf^ that in- 
clu'l'?i procr^s:* e\*ali«ntion, Lnipact evaluarion, and !$pccial studies. 
Spc-cial .;ti>(lies allow the decii^ion-miker to determine the need for various 
p-AijimiTu;. ij;f^^ict cvaluition serves to i»lentify those programs or projects 
which arc K-ivin?. t!iC qroritest (and poorest) qtialitative impact. Process 
cv:>lu3tion, on tlio other hand, is import.mt hecause it is necessary to 
know about tlie characteristics of projects that have a positive (or 
negative) iB^mct so that these projects laay be replicated. Witiiout a 
thorou^ knt^ledgo of project characteristics , it would be ii^xjssible 
to replicate successful projects in other Institutions and to avoid the 
failures that inevitably occos^iany the develqiment of a series of new 
projects. In addition, process evaluation allows piogmm nanagers to 
know if projects are carried out within the spirit pf pn^ras guidelines. 

Most evaluation systems us ctescribcd in tho literature include 
process evaluation as an integral part of the overall scheme* Kor 
instance, the evaluation model a^i developed by the Center for the StiK^' 
of Hvaluation (CSC) at UCIA includes the followijig evaluatic») coii|3on^ts: 

• needs assessment 

• pixtgram planning 

• in|>lciiKmtation evaluation 

• pro^'.ress evaluation, and 

• outcome e vol action 



Process rvalaitlon, in the CSR context, itKltidest both impXoixmtation and 
progress evaluation. A second ircijor evnluzttlon model as developed >y 
Daniel L. Stttfrtchcam is the CirP model tthldi includes as evaltttttlon 
ccsfionentsi 

• context e\'jilu:4tion, 

• input ov;iiuattcjit, 

• prtvjcsjs cvMlijntion, and 

• ppDtlurt r^vTiluation. 

In hoth of these systeins, proccfis evnl nation is as«;if9ied the role of 
rt'p^^itinM dutH to deci5:icifcnskcrs to crmtrol project operaticHis, 

has {ici^n pointed oiit in the literstiffe, process evaluation has 
three basic nlijectivcs: 

1. "tc detect or predict defects ii *jie procedural dcsi{»n or its 
inplomentation during the ii^iemc '^atton stap,es, 

2. "to provide information for prograjined decisions, and 

3. **to laaintain a record of the prc.odure as it occurs."^ 

these basic objectives are met td.th the raodel described in this r^rt. 
The mc^t^tcdolosical a^^noach described above tas bcai applied to 
a series of cducaticsud personnel training prograias in the U.S. Office 
of Hducatlon and the application of the process evaliration sefji^^t of 
the overall evaluation system is described in this report. 

Under the Hducatlon Professions OcvelopuKmt Act CEPBA) of 1967, the 
National Center for the l!q>rove!iient of Hducational Systcns (hCIBS), fomDrly 
the Bunsiu of Uducatioiial Personnel Dcsvolopaient CBHPIQ , of the Of fia» of 
Education, has boen authorized to :&ind a lar^ nuri>er of training pro- 
graats and projects desigiK^d to rcdicc the cr? rical shortages of many 
types of educatioiial personnel and to offer opportisiitios for isproving 
as well as reforming educational systems. These prograas cover a wide 
spcctruB of t^ education professions. 5^icclalists in early chilcfiKXxl 



I. *T;valuation t>i>es and a Model for B\'aluation/* Stuffleheam, 
Pro\ii5, fajtic, et al., in lUl ucntional itvatuation ml IV icisiomtakinj', 
Itiisca, Illinois, 1971. ***"" 



dowlo|wnt» .s^ieciril c»diicnti(?n, hillnj^l oducntlon, and vocttionnl 
eilMCntio:! have trainocU In nckfition, projects that foster educa- 
tional innovations such as Teaclwr Corps^ School Perstmncl lltillzatian, 
and Ihim/Ruitil School development haw been fiwded. Since the Center 
niK'.. ensure tlmt all of the diverse projects beincj ftmded are oporatinj; 
accor;l«nj« to pn»^,riim p.uiik>U:to< and are pm^ressinfi toward their objectives 
it \c:i' nwsjfrin' tV.it iheir --crivitles bo t:tc»nitored and evatimtfcl. Aca»iit- 
injjly contract wha in^MnloiJ t€> clfvc3o{» and itnplcvK-ni for the Center's 
piv».<»r;i« : -t pi\icess ovnhwtion system to assist decision-nsikers. 

T:ii<. r^jicirt cl'-niuncrt.s a t\%rt year eflr^rt tn develop and ir^hmmt 
a iiCi'. i -.'i i* ■iilujiJr.n r;yst#'m for the pvov.raffls of the National (x^ntcr for 
tr.e Tunrovonenr o( i:«ljicat ional Systons (NCIlfi) of the U.S. Office of. 
r:duc.itj«n. In tl*.e first year, the cv'alijation system was designed, 
questionnaires were dcvclqiod, and as:q*Aiti*r proorants were written. 
This system was then pilot-tented in 80 projects in eight NCTri5 prograiK; 
across the ccmto'. The sec(»id year's effbrt involved the full-scale 
i!H>lenentation of the systew in 438 projects in 12 NCIES progranB. 
The NCIfcS programs incltxiod in the study are: 

• BiUngiEtl Education 

• narly QuldlWHl 

• Hduc^tional Lcadei^hip 

• Piipil Personnel Services 

• School Personnel Utilization 

• S|;ecial Education 

• Teacher Corps 

• Teacher Development for Pesegreijating Schools 

• Traininj{ of Teacher Trainers 

• Urban/Rural School Devclopiaont 

• Vocatiraial Education Pari 552 

• Vocational Hducation Part S53 
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Mgtlicdologv 

The basic approach to the system de.<tign relics on obtaining certain 
basic information alxHit each progmra mid project. The logic for the 
process evaluation system is illustrated in Figure 1. Specif ictlly, 
each Center projjnflp has cle\'c]oped a sot of program guidelines (A) 
that projects; are expected to follow «md which, it is helieved, will 
result in tlic successful accoreil ishfrit-nt of ox'erall pro^raa ohjectU'es. 
Tne^o pro.'.ram f.ulfk'lincs provl<k> an owrail description of the goals 
and .i}f.ro'Jch<*.4 of iho projrrara as a v^Kile as well a«s providing a basic 

''.ir pr.,;'-r f .;..nvlornj^nt. In acklition, proRrara guideiinca contain 
a sc.-:.v; of iu\'.-.:-sn conditiai<; C«>, which are specific statenicnts of 
e.vjvctcfi project pcrfowifina*. 
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An cxiunrHc of .1 pronfitm cotnlitfon frm the Tonchcr Cotp* progitim Is: 

**lUiCh project will ck'VC'lop a <y!<tn!«tk- it)ftmi};ctncnt plan to fncllit.ite 
program development, cvalintion, am) modification.*' 
and from the Harly Chil(Sx)od ptt^ram: 

*TafticlpOTt5 i«hnuld ho pitjvidcd with qipommitiftn to cfevelop 
expertise In workittn wifh the (orwimity, identifying, undcrstmidlna* 
and vorkin;; within <*fcc'«;ion-r-iV»ntt stniCTure*,** 
tor each prf^ur.tnj corn >n, ciUiVititative inea.tures (B) were devflt^d 
ami ffvirwr.l and aiprov.^d t)y prop.rnn mfinnjiers. If data are cihtaincii on 
thfiisr^ nKTiiiitn**. of pro(>rain conditii^ii*, the pi.t^mn Anc^rial i.<t or 
evahnror cm {k't<'r!iinc nhr-hcr or not the con^Vitions are being met, 
Tiie';c i5if.a«iuros raipj^it take aijy foffltar frwa a siwplc Yes-iVo ref^Kmse 
f*1bs the project developed a systciflatic wanageincnt plan?*') to a 
nifltv»rical response (**InJicatc the nunher of participants; receivlnR 
practloM training.**) to a lengthy (^n-oniknl response (**tte!icribe 
overall project training strategici^.**) lAich would thon bo read and 
coded. Frcciuently, a nunib^r of data itcns would be roqaired to 
adequately ncanurc hour well a project wais addressing the data clement 
in qu*stion. 

Becau^ nuiy education pxogrui&s have a nuiiier of slailar cbjectivcA 
(e»g., the recruitment of minorities, cosnunity involvement, etc), and 
since it was necessary to gather similar bastoltne descriptive data on 
all proKraoit and projects, it was possible to construct a list of 
Contcr-wido common conditicms and measures (C). The questions designed 
to address these cotaaon conditions were placed in an overall questionnaire 
(D) with the data roqik^sted organized into seven basic areas: 

• inlbn»tion concominig participants, titolr dmrscteristics, 
and recruitment; 

• data on the length and nature of the training provided by the 
project; 

• a descripticm of the practiciw; 

• information concerning the amoisit and sources of funding; 
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• data*4i» projcHrt advijtoiy councils; * 

• project self-evaliKition information; and 

• a descriptltti of the dldtfeilmition of project infbfantian. 

ISiilo all education ptograw have amy siBilarities^ cn^ Center 
progrwa ha« he«i established to foster educational refona fron a different 
vienrpoint. Thiw» each program iKtd a nuni>er of j<uideliAcs which were 
program- icpccific. r^r tho.«e conditions thiit were profiraia-soecific (K), 
qunntitntivc mrafturcs were ckvcJopcd and placed in a series of projir.'«i- 
speci fie qiKsf ionnaircs (F), with a scpamte qticstiomutife for each of 
the tivrJve pn>grans Inrluded in the study. 

The next stt-p wns the one that caused the most di f f iculty fo/ the 
dcvelopHK^rt oi tlte evalmtion system, as prosram sanagers wore asked to 
establish i ?pct;ific goal (G) for each pro;rra» condition/seasure against 
idiich to gaujje project perfornance. Mi lie for the aost part 
flanagers wore reluctant to est^lish quantitative standards for the 
acasuroRcnt of project success or failure, they all uitlaately did so. 
It is believed that their reluctance to establish goals for project 
pcrfbnnaiKe was borne out of a fear that poor project perfonaincc 
would be interpreted as a failure on the part of tlw prognia Banag<;r 
himself. It was becaase of this factor that program managers were asked 
to establish the goals for their own programs. For Hvs mt part, the 
outside evaluator wno was thoroMghly fufldliar with the program cotAd lunre 
cstablislKid a set >f goals closely aj^roxiaatlng those developed by 
program managers. This, however, would have been nore m evaluation of 
the prograa as a whole than an evaluation of the projects. 

For nost of the prograa condition/neasure coBi>inati<ms, program 
MBiafors were able to establish a goal. In sone esses, however, it was 
either iapossible for the prograa sianager to state a goal as the 
progran condition/noasitfe coobination was descriptive in nature or the 
program unnager was unwilling to est^lish a goal. Khero a goal was stated 
by the prugnua, the data rqiorted by the projects wore ooaparod against 
the goal. feSiere no goal was stated, the project data were collared 
against the* progrfon a\\?rd>4C. Tor those projects cqtviUiig or exceeding 
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the thf data wrc presented a« reported. For thcu^Q proj.?ct who^ 

data were less than thn f*oal, the Jata wfc hij^ti{*htiHl ly placinft 
patvnthcitcii aroimd it, iiit: (UM'A). It should he noted at thU point 
that soBic projects here c^tahlishcd in order to address a specific 
need and thus may not have net all of the program jtoaU and yet be 
conjtiikMx*<I a Huccrsfful project. AdJitioiWlly, local i^ondlricni* in the 
fi«*ld ireiv hnvc prevent the .ittninmcnt of otu* or more of the goils l«y 
the projrc* rei;.irdle;>*. of the effort put forth. Xoncihclo<j<, the coricc})t 
of H pr^^^r.lr^/pr£>jcct. kO'iI aliens*** progr.mi ^peciaiijttjt ami evaluators to 
function hy rnanai5ffiK-*»i- hy-exctt>tion, concentrating their analysis and 
efforrs on those projects tJiat do not meet the proHraw goal. 

ExanH^lcr. of the prop.rain goal concept are that the Early Childhood 
proflr.181 set a RoaJ of having, participants spend at least SC percent of 
their practicum tine in contact with parotts ;«id tlie comntsii ty and 
the ^3ccial Education program sot a goal of having 100 percent of tiie 
participants receive a job in special e(&ication vpm project GOB|)leticm. 
Goals thus stated (in quantified tens) pemit an ohj /active analysis 
that would be less feasible with the more general i2<?d descriptive goals 
that are typically utilized in education prograas. 

Sunn&iri^ing the overall analysis ai^roach, thcn» actual perfomcmce 
can be confiared against planned perfbnaance as made explicit in quanti- 
tative goals for all conditions in the programs to form a discrepancy 
analysis 00. In addition, program statistical exports (I) and overall 
statistical reports (J) could bo produced fron the data base. 

In addition to the above surve>'s desij^ied to be sent to project 
directors, a separate survey Mas devfflopod to be sent to a saaple of 
participants in the projects surveyed. This participant questionnaire 
was designed to collect infoisation on the following areas: 

• expectations about the project, 

• participant cvaltAtion, 

• project activities, 

• project advisory coincil, and 

• areas of project strength and weakness. 



In addition, infomirion was collected on pnrtlcipmit demoaraphlcs 
(HKirltal ^rarust, age, .<eX| r^icc c*r ethnic haclt^sttntH), cthjCAtlon, and 
prior occi^tlon). Those data were collected to ncndt cross- ttfbiil at ioit^ 
to be developed in or<ter to detcmino idiether there t#as any si|;niflcdnt 
differmcc between demjgrai^lc ptimiui sid their respcmiies to the proje<:t 
f|iie«<ric^. Thiis, one could dctennine for a j*iven prftgmw, or for the 
Hurcati a^• a vvIjoIc, whether -^.it Is faction with the progran v.iried by 



In nil. a total of 14 quest icmna ires were ctevclopod ccm^ifttinft o£: 

• Part A - Comon ovt^nin questionnaire - (Project level) 

• Part B - TVelve separate program specific questionnaires - 
(Project Iwl) 

• Pai ticipitat quMtioniutire (IndivicUial participant) 

Thr prtHoct <{;io«5f innnaires wvrc iiwilrd to rill prt»j<*ct directors 
in the IZ prasiran^s in the imd<Ue of the project year* M overall survey 
rc.«ponr-p rate of 84 percent wnis achte\'ed for the 4.%^ project*? in the 
surv<»>, with of twelve pro^.mws nchi:»\'1ng over u 00 -percent 
rcspon-^e rate a.* nny be «jen in Tflbic 1 belcw: 

Table 1 

REFPCKSH RAir TOR P»Qlirr QUESTlCm^IRES 





Miffiber of 
Projects 


NuD6er of Projects 
Responding 


Nuflber 


Percent 


Bilini^jal Etktcatian 


13 


11 


85 


Early CSiUcSkkkI 


47 


38 


83 


Echjcational leadership 


IS 


14 


93 


Pu|ul fVrsonncl Services 


41 


57 


88 


School Personnel Utilization 


18 


17 


94 


%^ial UJucation 


39 


36 


92 


Teacher Corps 


70 


63 


90 


Teacher Dewlopment for 








IX>«ej»regating Schools 


43 


40 


94 


Training Teacher Trainers 


32 


29 


91 


Uri>an/Kural School IX^vclopismt 


24 


17 


71 


Vocational F^Jucation Part SS2 


18 


16 


89 


Vocational Hducation Part 553 


78 


49 


63 


Total All I'l-ograins 


438 


368 


84 



In cimtrast with tlic project qtif>5tLonnaire5, the parttcipmit qiies- 
tioitnain* was dUtrihutcd cbi a snwjile bait is wJth the queittlonnaires 
miled to selected project directors (projects were selected on a 
pit^ility proportional to size iWS) basis) « were asked to 
distribute them to participsmts according to a specific sinplo random 
sainple proce%1iires. Project dh-cctfirs x^vrc asked to return the cosplcte 
list of jKirticipnnis in tl.e proiict, indiciting those selected, thus 
porntttinQ verification tl.at the projKr selection procethjr^ was used and 
ffUowlnj; foJlow-tqi on na'i-re.vsraidcnts. An evaluation of the procedures 
utiUrecl and rlw? respc*n«-.es receive! indicated that a reprcscsntiitive 
sample oi' proy.mm participants was dniwn. 

The u«WT :>f tnis two-stapc s^fni»iing proct^din*e yielded an overall 
participant survey rc*;iirtnse rfite of S6 percent of the 3,2?:? participants 
in the r.;«itple that represented tJie total 27,41)6 participants in the 
programs for the year of this stuc^'. The rosponse rates by program for 
the participant questionnaire may be seen in Table 2. 

Table 2 

RKPO?^ lOTiS POR PAi?riCIPANT QUESnONMAIRE 





Participants 
' in Pn^rutt 


1 ^ 

Muriwr of 
Partici|^t5 
Sel(K:tct! 


Fartici|»ints 
Hi»HMn<iing 


Rate 


Bitin^ial tklucaticm 


408 


in 


113 


65.7 


Barly Childiiood 


M36 


SIS 


171 


54.6 


iiUtJcationiil Lca^kmsMp 


U9 


150 


96 


64.0 


Pi^il I^i^cnncl Services 


665 


1S2 


Ul 


73,0 


SchooJ t'cr^oimol titilixatiun 


2S4S 


456 


262 


S7.S 




S006 


S27 


194 


59.3 


Icachcr Corps 


2409 


258 


136 


S2.7 


Teachor I>civolui^eiit 


1S48 


279 


1S7 


S6.S 


for Ik^ogrcgatii^ 










Schools 










Training Teacher Traiiunrs 


30SS 


410 


211 


SI. 5 


Urhan/Rural 


20S7 


S58 


173 


48. S 


Vocational CUuMtlon 5S2 


2SS 


9S 


8S 


91.4 . 


Vocatic^al Kdticaticm 5S3 


7792 


305 


136 


44.6 


Total - 12 Center Pra/jnurs. 


27496 


S27S 


1845 


S6.5 



eery ftvfiiu^ 

CoL i;} utcf Su ftynre ** 

Because of their vastly di ffcroit luiturc, two sciKirntc ccofiutcr 
softw.-m* s>'stOfiis wero utilized for tlie project nnd participant 
qtt£!5tioimHires. The participant qiic5tionnaire was developed to be analyzed 
in a standard cross- tab fornnit and hrnce v.'as processed by a ccmwercial 
co»!ijM«tcr fii-ir. Tlic project qiK-stloniiaires, on the other linnd, were 
highly sjH^cl.'ii^^od both in their dcntQn uhd in the typoa of anfiJyj;c.< 
l.«Mf Uv«'.t;J nnpi ictl, lii'ucc a syiSto - of specialized proftrarv? icas 
drr.Mop'.v.:. Yn\^ ."systrrj cojxists of :i collection of CCWh lanRiWRe 
pr'.'.:; 'H'K ftWi riT.cd fo he iV4iirKitiblc with the conpiiter system) 

mil ay^u-n ivjrt:". It wns dcsiiinc'l for and inq-»]cinented on the IPM 
Sy--ttrn 3fif)/N:-ock-i 65 and "-TOAbdcI IT.S iLsin.a the tm distributed OS-NVi* 
Sorr.vare r/vr^uiter System nnd required the folioi^inj; peripheral caqiiipisirnt: 

• an Jm on-line 1403 printer, 

• an JKV on-line 3S40 card rca<fcr, 

• at least one I EM 2314 disk storage .tevice, and 

• at least three IBM 729 ma.qnetic tape units. 

In addition, this system can also be loaded and executed fn» a 
remote job entry terminal such as the Data 100 or I»f 2784 RJE terminal. 
l"ho systoiB was designed to produce a{"»iiroxiittately 47 predesigned td>les 
in 31 different presentation fonnats in a ^generalized inanner and 12 
proiiraw discrepancy analysis reports. The PPQ-DTS was intent icHially 
designed in an c^n-ende*d and general fashion so that additloml coding 
or totally new output table generation procedires could be added to tho 
system fr<unework in a timely and easy ^nner. 

Structurally, the programs can be ccmsidered to consist of various 
indc]>endent operational siodulcs that perform sane function in the 
questionnaires during this flow throi^i the system strsaa. At the highest 
level of consideration, the system ccmtaias three major modules. Ihsfy 
arc: 

1. Editing and ntuintcnancc of the quest lonnain; data base, 

2. Rc{>ort printing file prci^aration, and 

3. G(»icration of the required tables, 

* « 
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Within ^:tch of theftc rarxhtles, i^ortwan^ proi'i'asH jx?rfor»!i prescribed 
operations on the data. Ituring oxmit ion of tlic <»dit ntodulc,' the edif 
pitigraB perfoms various niineric and syntax data diecks. further 
doimstrcamy titc if^date progttm of the ataintcnance ttodulo not ott'v 
prDcliices data base records but also cxccutOis user-directed ta range 
checV.s. Still fiirthcr clom'^trcnn, records are extmct'^l, ;««Tk»d, :md, 
flnnlly, in tVr Inst moctnlc, dnta rrf" prcf*er!ted in e>i*n!it tanlo report"*, 
Bet.i 'ise .'>r t\.o f:tf^d>or of &iiH fields incliktcd find le r*i»!t»er of projects, 
ta->«»\'*., \>rs'pjtm i»onl coriparlsons, a syst''"" -^torngp capacity of at 
Jeu ;l 2r*0K is -cc^tii red for the cmjjlcte v\r, lithntioji iwodules such as 
€idit» tj]AL'itc, oi.c. ifViy he ;iccn»rpU$b'''' vitj^ imK'Ji loss storfipiC capacity. 

A sir^pi i Tied Clew diri^jraia lo^* the software system nay be seen in 
>ip.ure 2. 

It should be noted thnt ,e cxsnfirehonsive fiatka^e of canjyjtcr 
ptOK7-an» wen; developed i'luse of tlic large niisber of projects surveyed 
and tlje \'c>ry large Ir t.»ai»nts that were developed. For a smll nini^er 
of projects and fhr .nstnancnts that request a swall nuRl)er of data 
iteins, hand piv^-cisint* wuuld be feasible and ii^ld result in a significant 
cost savin.-- Mie great flexibility designed into the software system 
describe* cibovc would be lost, Iiowever, and this imist be wci^ied in the 
disa Ion as to the process Ing mode to be adc^tcd. 
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j^ alysi.< and Prey on t at i on of Datn 

Hie ck?vplo|>nient of the analysis plan for any evaluation systcni imist 
precede the development of the instruments and the sample. Hits sequence 
Is essential if data are to be collected correctly. The dcwlopnent of 
mstruracnts prior to tlic existence! of an analysis plan uill result in 
datM ftroni^s or the collection of oxtn?ncoi» dit.i. Hum the ann lysis pl.'m 
K.'is p repair:! very eiirly in I he UevclojTnent of this sti^Jy. 

Ti;c b.M:;jc approach lo tlic <lo.^ ii»n of the analx^is plan was to de]ifnd 
hon'iJy on the needs of efliicational decision-makers t<» clutcmine the 
r.p-.^cific an.iJvsis proceilnros. Jt is believed that the ultimate use of 
the data slio'^ld he the diivinf, force of the analysis ratlicr than the 
fonnat and form in which it i^ most convenient to collect it. If the 
data collected are not pi^linhcd in a £om usable by various levels 
and tApes of dec is ion -makers, their ultimate piirpose is seriously diluted, 
lliis an^ronch was inherent in the overall system design, relying as it 
did on program conditions and goals. The analysis for each of the 
three types of questionnaires, while interrelated, was developed along 
separate lines. 

As described previously, the data collected on the participant 
quest ioimai res were processed in cross- tab format providing sunanaries 
of the data in a series of approximately twenty tables. The data were 
sifflsiutri^ed both for each individual program and for the Center (all 
twelve prof,ra!iis) as a whole. The Center data were processed using the 
program as the basic variable while the basic variable for the program- 
level data was a series of participant demographics (age, race, sex, 
education, and prior occupation). Ihe basic question about \ihich data 
were analyzed were jwrticipant expectations about the project, partici- 
pant evaluation, project activities, pi*oject advisory council, and areas 
of project strength and weakness. A samiJle of the data output may be 
seen in Table 3. 

The Part A questionnaire data were processed in a descriptive mirnner. 
The goal with this data was to present statistical descriptions of the 
piojects, programs, and the Center as a whole. It was designed to 
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piwiUo informtion to <kH:i?ioii-m.ilicrs in sewn basic arcnm Snfcroatioii 
concerning participantn, their dm racier is tics, and tl^nr recruitment; 
data on the length and nature of the training provided the projects; 
a description of the practioBa; infonaatioi concerning tho anoint and 
soiircojs of ritri<Jinss; and <lescriptive data ccmcemins; ad\'isory councils, 
project f-rva lu.tr i«-^n» anJ the dinsominntlon of project infoimition. 
!r iJ05rir.n to pmvi.IInu :» tect iptive rt'port conccmitiji proiect :ictivJties, 
*\,'iS i,]yiYo wh nlUnv^ t-cc ij^ton-ti^kt rs to bo^jhi to view tlx* qiuntitntivo 
Ut< cr->\<:^<.^<\ to the qiirilit.ith'r) iii^.)!ict of tlie prf>ftr.iins. Data concdMiriK 
llv n'.rhcr Til cb:!r.-ict<^ri';tk-.< of tr3iTKH?s nn^l thf* .iflr)i»it an<I type.-* of 
tf.iiTsin.i ].'.-r:.'i\ u nx-nx'.irv oi the qinnt i tati vc in^vict that the prop.mn! 
vUl l.ivo. A .--..'iKfilc nf the (^itptJt frnm the hasir statist icnl packn^^ 
i-wy 1>e sorii lA Tr'jhlc 4* 

hhilc tlie participant questionnaire datit wrc cror.s- tabulated and 
the statistical data reported in a series of tables designed to fill 
specific information need.*, the discrcpanc)' analysis dita were treated 
soaicwluit differently. /\s described above, each prograia manager was asked 
to specify a goal for each origination of prof.nBR condititms and quanti- 
tative measitrcs. For most of the contoinations, prograas established a 
goal to assess the successful perfonaancc by a project in process terns* 
Where a goal was stated hy the program, the data reported by each project 
were confiared against the goal. Where no goal was stated, the project 
data were conipared with the program average. Mien 'he data reported by 
a project did not equal or exceed the gcml established by the program, 
the data wore highiightcHl by placing parmtheses around It, allowiiig 
analysts to quickly determine where assistance was needed by the projects. 
The data, then, were reported on a program basis wi^ tho pxoject serving 
as the basic unit of analysis. A sai^le of tho data output that was 
produced may be seen in Table 5. 

In addition to tho project-by-pro jcct rejK»rting of the <kta, two 
typca of summaries were developed. The first simaary was the Project 
Dlsctxtumcy Analysis Sunsnaxy. This suHiaary was designed to indicate t!ie 
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percent of the propmm conc1itionsA|imtit.'itivc ncnj^iircs th:it each project 
.success fuJ ly iiK't as cstahlishctl by tin* ])roiji;ua i',o:ils. h!iiJo each pni^^rnn 
coiKlttJon does not necessarily c:irr>» an equal h-cigl^t with progrom iaan.'ij;crs 
in general it uuiy be said that the hij;her the ntsrijer of c<»iUiti«is met, 

thv liore "^'Acos^ I>il the project v.ns in M-otini' the ]iroj»nini objectiws. A 
sc»-ii.-^ of t.ibic>. r;h<>,\'!nu ^hv i*:iiiktn»« cif projects in each pro.ur.un wns 
ucvc !o'/.eu. Th<' utj'llty oi this SJmsT.Mn' ^as that it permittcvl proi»nnn 
r.'AH; .-.»*• IS to Ci'-.ick'.y ucter.iliio which oC ihes^* projects wrrc ^h^ wenkcst 
iji t:M';'.s of rn-'^ring prvv..r;ji'i gonls. Iv shmild be re^iK^^^vered at tins point 
thJJt in onler to n::ji<e fina] decisions concern inp, the |x-rfoTJi^ance of « 
i^'iVvw jirnHTi-. it \.< not sisff Icient to rely solely on the projects 
inJc'v uf upcras i<^n.'j| CGnr.i"^tenr>* but is necessary to coml-iine this with a 
kncwlfd.oe cf the f special) objectives of the project, the cnvironnient in 
which it me^t ftmction, and other information concerning Xha project's 
effectiveni^ss. 

Hie second sumnan* was the Measure Discrepancv' Analysis Suinaary. 
llii:* surereiry indicated tlie percentage of projects that success fuHy met 
each condition/ measure and thus was indicative of how successful the 
total projjnun was in meetinjj its own self-established jjoals. 

rinally, cis a check on the ability of the mail questionnaires to 
collect accunitc data, field site verification visits were scheduled to a 
small sanfile of the projects. The aj^jiroadi taken was not only to in- 
dc-jx-nilently collect the data requested in the questionnaire £rm the 
basic data sources av;i liable to responvlcnts, but also to request 
similar pieces of data fivm a variut)* of sources. Mvmg tlie imlividi&Jls 
that could be in televiewed at a given site were the project director, 
project staff, i>articii>ants, college representative, school representative 
and coJwjiu]\lty representative. Since in most cases the project director 
provided the data reported on the questionnaires, the verification 
directed to him co,..emcd his interpretations of certain questions and 
the general proble»is ho had in resiwnding. Other tyi>es of persons were 
selected to provide other views of the questionnaire information and 
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in mny casci*, nxirc than one T>i^e of jiorson w.is intcrvitncfHl for o given 
question to ho VLrifictl. llie results of the valiJIty stutly Imllcatrd 
th:it pivjcct directors n-cre able to provide tho t)'pc of information 
miujstcd aiiJ tliat the data provicted were accurate. 
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The system as d!5ctib«!feil jiIkivc, ulnlc j^kjU in its ability to collect 
both UcM-riptive ami piw sj» cval nation data, docs h.ivi? some linitationH. 

t. Questionnaire design is not a onc-tiuip occurranco, (Jupstien- 
iwircs imist be revis'.nl .nnntwlly 'j< pmjjmm j^nck>lin<>s arc chflnged. 
l^ctvniT tl'*^ P.irt \ rn- oT:?on iT'jsiniir.ont wis <lc<iiycd to {»c jjencn^t ami 
to * r.>-.--o\v ail ]♦••«• •i'.n«<, it tsxcI little ir.cbtinj* on an M-imml 

K'Ki.-s. ii is iK»f r»ie c:«sc* f^>i* the prajjr-a^- specific Vit^stionnaircii, 
Tlt'.t.'.' qV'.-it ioMiairrs .irf« fkrsi>^-...£l to rollci't dita that specifically 
rrlr.t:- to t],n ;ii«!iv;.'!'.nl pfop.fJJiu ;;nd thiiis Srtist he mis^ii as proj^nmi 

2. if no prostr.:.a descriptiL-a*? exist prior to tho start of tho 
cvaltntlcn, tlicy mat ht? ik!Vt!loi:ccI ai\ a fir.nt step. .\s the Rtiideliiier* 
are juviiarcU, pro>irar.i amditions uUI also be develojvid. Tliis exercise 
rcqtalres a siijiificant .iwoiait of tiiac but It is highly useful in that it 
forces pri;;rajn manngcrj to think throniih the overall goals and objectives 
of the projirasi and to state tlicse explicitly. 

3. Specific goals for each of the prograa conditions sust be 
developed. Khile ilK*sc could developed by an external evuluator, 
they will be C'jnsi<k*rably more sipiificant if established by the program 
luana^er. The difficulty arises in that aanaRers often view each project 
as totally unique and are reluctant to sec the comon, program- related 
threa<f in all. 

4. Ihe standard timing prc^lcnts inherent in all data collection 
exist heu*. Project activities occur over some finite tiae period but 
the MUwfSiioiiiviLrcs Mtst bc^ coe4)l<^ted by pmJcHit directors at a glvi^n point 
in ttau. Distribution early in the >ear forces project directors to 
estiwito future activities based on tlielr oiieration plan and curreitt 
activities hfiere^m distribution late in tho project year results in the 
collection of njtrus|>cctive diitfi, mui?h of which mii»|it iK>t have been 
recorded. Hiven the lliititutlons of both, coltecticm relatively early in 
the year resuins the better with the najor achrantage beins that tlio data 
can bo collected, analyzeti, and reported to program managers cturly enough 
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thit corivctivc action cnn ho taken and projects that are not atlhcrlnfl 
to prci;'.r;jtn guicV} litcs rrtlinvtCHl. 

S. Gmtt caro mt^t he vxercis^ to assure that the sy^tcn is 
not used to collect too tatidt data. There is a great tendency Cor evalta- 
tor.4 and prr»;:ra!3 i'i.maft<T!< to cf»llcct dafrt that would he •*nice to know** 
but to; whkr, t'....!'.- i;; tiot a fie ni'CkJ. Kith a svftren as ]Mtvi'orfiij a? 
thi-* n! J '.■!]t':; t't.c rn:ir oT col Joctlii*'. an.l prf.:i SHii!R .-iddir ioiial 

dti?i u.'.*-*.. J.-4V, it u vers' {V!A/ fiir thiis to ncnir. Vhc results of this 
i'rv t':-:! .SiC tIritM ri'j'nrfina birr.rifn on rw-*[Oi»ilcnt.^ i.-; ;;fnni namtly 
incrc.f .. J, {"rcviiir'n* tlrc-ron^Onji tic valtdify c»f tl.o data reported, Jind 
that r'f • '):-.,i;r.i:^ ; nMitrr rccclvcj* n^MC d-ira than he Is vfillinn lo 
rcvtcM ii.u*. iij'.aly-i'. 'i'hf> ptf)}<lcm mv he .u-nidevl 3f the anaJysisi plan 
i.*; de.; i -rc*d ;iniJ .i.Ihorcd to he- fore the in.itnanwits aro dosisiied- 

In addittcm to i-af.ic refincaents to pcndt the system to function 
more siatwthly, two furtl.cr stei-:; should he taken. 

1. disciiiird previciisly the goals as developed for tho projects 
did not noccssarllv earn* an cqial wei|»ht with prograa mnagers yet 

thc>' woitJ treated a^ If this wore true. A weightinft schcat needs to 
he developev! in conjunction with prc^gram managers and :^lied to all goals. 
The use oC thi;; Kcifthting scheme Msuld permit a trie Index of operational 
consistency to be developed th<'it would have some qiKsitifiable meaning 
in an oveiali evaluation sense. 

2. Both the goals and the results of project porfdneiKe evalua- 
tion must bo stated to pix>jects. t^lle it is true that the program 
condltJuiis arc explicitly stated in the guidelines developed hy each 
pit>gmii and distributed to each grant applicant, the indication of 
exiiected minimuii t»crfonmince would servo to set a standard for projects 
to atte«ii|it to meet. I'inally, the results of the evaluation should be 
fmnL'diatoJy forwarded to the project directors to allow th<»t to judge how 
their project is jwrfonaiug lelative to tho level of performance 
considered satisfactor>'. 



22 



r.:sT COPY m\M£ 



'flic mrmafcmrnt and cv.i)tt:itlon of education pro^niins rciquires a 
timely, accurate infomition system. The d}scrct>an<y analystift oodel 
des?a ibtHi aid.* in providing this infomntion \n a format that nay be 
[♦roccsjsv 1 and yi^^lit^i the methrv*] by vfhich siidi a styj^tcw way be developed 
ami it"!» lemon ted. liu'ijdin;; ft\%} spc»cific i^tatetnents of ex^iectecl project 
|icrforif'.|*,c^» ih1< sruiy shcn\'etl hOK- ^.iie«*tioniHiTf.>? wiy f»e devel^^-ted, 
awil/si.' t>i.*ins f rt..pi>i*ed, rnd annlytical reports pTVcwKrod to fiwhle a 
revicrtvr ui quickly ienrn the i»nitus of an education pn).iect. Thic 
<v.<tcin . rovM.^s th-i hi'sic ty,vs of dnta f<^r dccision-mnkers: basic 
ck'S'Tiv-U ;v inrGiM.'it ion 10 jionnit stati.^tical sjtiiaarics of pr^ior.r.n oper^ 
ti:-n to ho pit^l'Jir.^d md i-evi«\«cd, and operational consistency data which 
ajlow tlcc!Aion-in.'iki'« to revieiv project operations jind to fmcticn by 
itimwftciacnt-by-cxceptioji, concentrating their efforts on those projects 
most in need of tedmical ossi^t^mce. 
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